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On the nonnegativity of L{^,tt) for so2n+i 

By Erez Lapid and Stephen Rallis* 
Abstract 

Let vr be a cuspidal generic representation of S0(2n + 1, A). We prove 
that L(|,7r) > 0. 

1. Introduction 

Let vr be a cuspidal automorphic representation of GL„(A) where A is 
the ring of adeles of a number field F. Suppose that vr is self-dual. Then the 
"standard" L-function ([GJ72]) L{s,'it) is real for s G M and positive for s > 1. 
Assuming GRH we have L{s,tt) > for ^ < s < 1, except for the case where 
n = 1 and vr is the trivial character. It would follow that L(^, vr) > 0. However, 
the latter is not known even in the case of quadratic Dirichlet characters. In 
general, if vr is self-dual then vr is either symplectic or orthogonal, i.e. exactly 
one of the (partial) L- functions L^{s, vr, A^), L^{s, vr, sym^) has a pole at s = 1. 
In the first case n is even and the central character of vr is trivial ([JS90a]). 
In the language of the Tannakian formalism of Langlands ( [Lan79] ) , any cus- 
pidal representation vr of GL„(A) corresponds to an irreducible n-dimensional 
representation ip of a conjectural group Cp whose derived group is compact. 
Then vr is self-dual if and only if (p is self-dual, and the classification into sym- 
plectic and orthogonal is compatible with (and suggested by) the one for finite 
dimensional representations of a compact group. Our goal in this paper is to 
show 

Theorem 1. Let n be a symplectic cuspidal representation o/GL„(A). 
Then L(i,7r) > 0. 

We note that the same will be true for the partial L-function. The value 
L(^,vr) appears in many arithmetic, analytic and geometric contexts - among 
them, the Shimura correspondence ([Wal81]), or more generally - the theta 
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correspondence ([Ral87]), the Birch-Swinnerton-Dyer conjecture, the Gross- 
Prasad conjecture ([GP94]), certain period integrals, and the relative trace 
formula ([JCOl], [BM]). In all the above cases, the L-functions are of symplectic 
type. Moreover, all motivic L-functions which have the center of symmetry 
as a critical point in the sense of Deligne are necessarily of symplectic type. 
In the case n = 2, tt is symplectic exactly when the central character of tt 
is trivial. The above-mentioned interpretations of L(^,7r) were used to prove 
Theorem 1 in that case ([KZ81], [KS93], using the Shimura correspondence in 
special cases, and [Guo96], using a variant of Jacquet's relative trace formula, 
in general). The nonnegativity of L(^,7r) in the GL2 case already has striking 
applications, for example to sub-convexity estimates for various L-functions 
([CIOO], [IviOl]). Wc expect that the higher rank case will turn out to be useful 
as well. The nonnegativity of L{^,x) for quadratic Dirichlet characters would 
have far-reaching implications to Gauss class number problem. Unfortunately, 
our method is not applicable to that case. 

The Tannakian formalism suggests that the symplectic and orthogonal 
automorphic representations of GL„(A) are functorial images from classical 
groups. In fact, it is known that every symplectic cuspidal automorphic repre- 
sentation TT of GL2n(A) is a functorial image of a cuspidal generic representar 
tion of S0(2n -|- 1, A). Conversely, to every cuspidal generic representation of 
S0(2n + 1, A) corresponds an automorphic representation of GL2n(A) which 
is parabolically induced from cuspidal symplectic representations ([GRSOl], 
[CKP-SSOl]). As a consequence: 

Theorem 2. Let a be a cuspidal generic representation of S0{2n+1, A) . 



The L-function is the one pertaining to the imbedding of Sp(n, C), the 
L-group of S0(2n + 1), in GL(2n,C). By the work of Jiang-Soudry ([JS]) 
Theorem 2 applies equally well to the completed L-function as defined by 
Shahidi in [Sha81]. 

We emphasize however that our proof of Theorem 1 is independent of 
the functorial lifting above. In fact, it turns out, somewhat surprisingly, that 
Theorem 1 is a simple consequence of the theory of Eisenstein series on classical 
groups. Consider the symplectic group Sp„ and the Eisenstein series E{g, ip, s) 
induced from tt viewed as a representation on the Siegel parabolic subgroup. 
If TT is symplectic then for E{g,ip,s) to have a pole at s = | it is necessary 
and sufficient that L(^,7r) ^ 0, in which case the pole is simple. In particular, 
in this case e(^,7r) = 1 by the functional equation. We refer the reader to the 
body of the paper for any unexplained notation. Let ^p) be the residue 

oi E{-,(p,s) at s = ^. It is a square-integrable automorphic form on Sp„. A 
consequence of the spectral theory is that the inner product of two such residues 
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is given by the residue 9Jt_i of the intertwining operator at s = |. Thus, 9Jt_i 
is a positive semi-definite operator. First assume that the local components 
of vr are unramified at every place including the archimedean ones. Then by 
a well-known formula of Langlands ([Lan71]), the intertwining operator 
satisfies 

Tl{s)vo = L{s, vr) /L{s + 1, tt) • L(2s, vr, h^)/L{2s + 1, vr, A^) • vq 
for the unramified vector vq. Therefore 

m-iVQ = \- l(\,tx\ /l(^,-k\ ■ reSs=iL(s, tt, A^)/L(2, tt, A^) ■ vq. 



Since L(s,7r) is positive for s > 1 and L(s,7r,A^) is real and nonzero for 
s > 1 we obtain Theorem 1 in this case. In order to generalize this argu- 
ment and avoid any local assumptions on tt we have, as usual, to make some 
local analysis. For that, we use Shahidi's normalization of the intertwining op- 
erators ([Sha90b]) which is applicable since vr is generic. Let i?(vr, s) = R{s) = 
®yR^{s) : /(tt, s) /(tt, — s) be the normalized intertwining operator. Here 
we take into account a canonical identification of tt with its contragredient and 
suppress the dependence of Rv{s) on a choice of an additive character. Then 
dJt{s) = m(s) • R{s) where 

L(s,7r) L(2s,7r,A2) 

Tfli S ) = • 

^' £(s,7r)L(s + l,7r) £(2s, tt, a2)L(2s + 1, tt, a2) ' 

Hence, 9Jl_i = m_i -i? , where m_i is the residue of m{s) at s = ^, and the 

operator R{^) is semi-definite with the same sign as m„i. On the other hand, 
the argument of Keys-Shahidi ([KS88]) shows that the Hermitian involution 
i?(7r^,0) has a nontrivial eigenspace. The main step (Lemma 3, proved 
in §3) is to show that R{'Kv, 5) is positive semi-definite by "deforming" it to 
i?(7r^,0). This will imply that m_i > 0, i.e. 

^{h^) res,=iL(g,7r, A^) 
^(3^^) ■6(l,7r,A2)L(2,7r,A2) > " 

Similarly, working with the group S0(2n) we obtain 

reSs=iL(g,7r, A^) 
e(l,7r,A2)L(2,7r,A2) 

if TT is symplectic. Altogether this implies Theorem 1 (see §2). We may work 
with the group S0(2n-|- 1) as well. Using the relation £(i,7r(gi7f) = 1 ([BH99]) 
we will obtain the following: 

Theorem 3. Let tt be a self -dual cuspidal representation of GL„(A). 
Then £:(^,7r, A^) = £(i,7r,sym2) = 1. 
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This is compatible with the Tannakian formahsm. In general one expects 
that e{^,Tr,p) = 1 if the representation p o tp is orthogonal ([PR99]). This is 
inspired by results of Frohlich-Queyrut, Deligne and Saito about cpsilon factors 
of orthogonal Galois representations and motives ([FQ73], [Del76], [Sai95]). 

The analysis of Section 3, the technical core of this article, relies on de- 
tailed information about the reducibility of induced representations of classical 
groups. This was studied extensively by Goldberg, Jantzen, Muic, Shahidi, 
Tadic, and others (see [Gol94], [Jan96], [MuiOl], [Sha92], [Tad98]). 

Note added in proof. Since the time of writing this paper Theorem 1 was 
generalized by the first-named author to tensor product L-functions of sym- 
plectic type ([LapOS]). Similarly, other root numbers of orthogonal type have 
shown to be 1 ([Lap02]). 

The authors would like to express their gratitude to the Institute for Ad- 
vanced StTidy for the hospitality during the first half of 2001. We would also 
like to thank Professors Herve Jacquet and Preydoon Shahidi for useful dis- 
cussions. 



2. The setup 

Let F be a number field, A = Ajr its adeles ring and let tt be a cuspidal 
automorphic representation of GL„(A). We say that tt is symplectic (resp. 
orthogonal) if L'^(s, 7r,A^) (resp. L'^(s, vr, sym^)) has a pole at s = 1. If vr is 
symplectic or orthogonal then vr is self-dual. Conversely, if vr is self-dual then 
vr is either symplectic or orthogonal but not both. Moreover, if vr is symplectic 
then n is even and the central character of tt is trivial ([JS90a]). Our goal 
is to prove Theorems 1 and 3. In this section we will reduce them to a few 
local statements, namely Lemmas 1-4 below which will be proved in the next 
section. They all have some overlap with known results in the literature. We 
first fix some notation. By our convention, if X is an algebraic group over F 
we denote the F-points of X by X as well. Let J„ be n x n matrix with ones 
on the nonprincipal diagonal and zeros otherwise. Let G be cither the split 
orthogonal group S0(2n -|- 1) with respect to the symmetric form defined by 

/ Jn\ 
1 

V Jn J 

or the symplectic group Sp„ with respect to the skew-symmetric form defined 
by the matrix 

(a Jn\ 

\-Jn J 
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or the split orthogonal group S0(2n) with respect to the symmetric form de- 
fined by [ "n* 1 • Then G acts by right multiplication on the space V of 

row vectors of size 2n or 2n+l- Let P = M -U he the Siegel parabolic subgroup 
of G with its standard Levi decomposition. It is the stabilizer of the maximal 
isotropic space U defined by the vanishing of all but the last n coordinates. 
We identify M with GL(V/U-'-) ~ GL„ where U"*- is the perpendicular of U in 
V with respect to the form defining G. We denote by i' : M(A) R_|_ the ab- 
solute value of the determinant in that identification. Let K be the standard 
maximal compact subgroup of G{A). We extend u to a left-?7(A) right- K- 
invariant function on G{A) using the Iwasawa decomposition. Let Sp be the 
modulus function of P(A). It is given by 5p = i^", i/""*"^ or according to 
whether G = S0(2n + 1), Sp„ or S0(2n). Let vr be a cuspidal representation 
of GL„(A) and A{U{A)M\G{A))Tr,s be the space of automorphic forms ip on 
U{A)M\G{A) such that the function m {rn)5 p{m)~^ /"^ firnk) belongs to 

the space of vr for any A; G K. By multiplicity-one for GL„, A{U{A)M\G{A))t^^s 
depends only on the equivalence class of vr and not on its automorphic realiza- 
tion. By choosing an automorphic realization for vr (unique up to a scalar), we 
may identify A{U{A)M\G{A))t^^s with (the K-finite vectors in) the induced 
space /(tt, s). The Eisenstein series 

E{g,'f,s)= ^ <yf (75)^^^(75) 
7eP\G 

converges when Re(s) is sufficiently large and admits a meromorphic continua- 
tion. Whenever it is regular it defines an intertwining map A{U (A)M\G(A))^^5 
A{G\G{A)). It is known that the only possible singularity of E{g, ip, s) for 
Re(s) > is a simple pole at s = | (except when tt is the trivial character and 
G = Spi, where there is a pole at s = 1). 

In the case G = S0(2n) let S be the outer automorphism obtained by 
conjugation by the element 

/ In-l \ 

1 

1 

V In-l / 

of 0(2n) \ S0(2n). For the other groups let S = 1. In all cases we set 
^ = S". Then 6 induces the principal involution on the root data of G. 
Note that {P, 0{P)} is the set of standard parabolic subgroups of G which are 
associate to P. Fix w € G \ M such that wMw^^ = 9{M); it is uniquely 
determined up to right multiplication by M. Let ^ : M 6{M) be defined 
by = wmw~^. Denote by w-k the cuspidal automorphic representation 
of 9{M){A) on {(p^ : ip e V^} where (^"(m") = (p{m). The "automorphic" 
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intertwining operator 

m{s) = m{7r,s) : A{U{A)M\G{A))^,s ^ ^(^(?7)(A)e(M)\G(A))^^,_, 
is defined by 

[^}}l{s)(p]{g) = / (p{w''^ug)u^{w~^ug) du. 

Let E^i{»,ip) be the residue of E{g,ip,s) at s = ^. It is zero unless wtt = it, 
and in particular, 6{M) = M, i.e. 9 = 1. The latter means that P is conjugate 
to its opposite. We say that tt is of G-type if E^i ^ 0, or what amounts to the 
same, that Tt-i ^ where DJl-i is the residue of SDt(s) at i. In this case 
defines an intertwining map A{U{A)M\G{A))^ i AiG\G{A)). The inner 
product formula for two residues of Eisenstein series is given by 



(1) / E_iig,'Pi)E-iig,'P2) dg 
Jg\g{a) 



1 1 

Jk J a 



^!0t-i(pi{mk)(p2{mk) dm dk 

M\M(A)i 



up to a positive constant depending on normalization of Haar measures. This 
follows for example by taking residues in the Maass-Selberg relations for inner 
product of truncated Eisenstein scries (cf. [ArtSO, §4]). Alternatively, this is a 
consequence of spectral theory ([MW95]). 

We let tt'^ be the representation of 9{M){A) on 14 defined by Tr'^{m^)v = 
Tr{m)v. We may identify tt" with wtt by the map (p i— >■ (pK Let M{s) = M(7r, s) : 
/(tt, s) -f(7r*, —s) be the "abstract" intertwining operator given by 

M{s)ip{g) = / (p{w~^ug)v^{w~^ug) du. 
Je{u){A) 

Under the isomorphisms 

A{U{A)M\G{A))^^s - /(7r,s)and 
yl(e(C/)(A)e(M)\G(A))^,,_, ~ IiJ,-s), 

2Jl(s) becomes M{s). 

Let : M M be the map defined by m^ = 6{m'^). We will choose 
the representative w as in [Sha90b] so that when M is identified with GL„, 
becomes the involution x i— > *x~^Wn where 

v._/ (-ir if^ + J = n + l 
ywnhj <y Q otherwise. 

In particular does not depend on G. A direct computation shows that 
(2) 

w'^ G M corresponds to the central element (— l)"(resp. (— of GL„ 
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if G is symplectic (resp. orthogonal). We define and tt^ as before. Since tt 
is irreducible we have ([GK75]) 

(3) TT^ is equivalent to the contragredient tt of tt. 

Thus, for vr to be of G-type it is necessary that 6 = \ and that tt be self- 
dual. If TT is self-dual we define the intertwining operator l = Lt^ 
by = . It is well-defined by nniltiplicity-one and does not depend on 
the automorphic realization of vr. Wc write i{s) = /.(vr, s) for the induced map 
/(tt^s) Ii7r,s) given by (g) = L{f ig)). Note that when 6 = 1, l{s) 

is the map I{tt^, s) /(vr, s) induced from the "physical" equality of the two 
spaces A(U{A)M\G{A))^uTr,s and ^(C/(A)M\G(A))^,s. Assume that tt is self- 
dual and that 9 = 1. Then as a map from /(tt, s) to /(vr, —s) the intertwining 
operator 9Jt(,s) becomes i( — .s) o Af (,s). Let (•, ^tt be the invariant positive- 
definite Hcrmitian form on vr obtained through its automorphic realization. 
This gives rise to the invariant sesqui-linear form (•,-) = {■,-)s '■ /(tt,— s) x 
/(tt, s) — > C given by 



{^1,^2)= / {(pi{k),(f2{k))Tr dk. 

Thus, the right-hand side of (1), viewed as a positive-definite invariant Hermi- 
tian form on /(vr, is — o M^itpi, 992) i • 

In the local case we can define tt^^, tt.J' and the local intertwining operators 



M^{s):I{7r^,s)^I{7Tj,-s) 

in the same way. Fix a nontrivial character ip = ^vi^v of F\Af. For any v 
choose a Whittaker model for vr^, with respect to the ''-stable character 

/ 1 Xl * * \ 



1 ■. * 

1 Xn-l 

V 1 J 



1pv{xi + ■ ■ .+Xn-l). 



If TTv is self-dual then we define the intertwining map t„ = I'tl^ : tt^ by 

MW)] ig) = Wig') 

in the Whittaker model with respect to By uniqueness of the Whittaker 
model Ly is well-defined and does not depend on choice of the Whittaker model. 
If we change ijjy to tpvid-) for a e F* then ly is multiplied by the sign a;^~^(a). 
If TTy and ipy are unramified then ty{u) = u for an unramified vector u since 
the unramified Whittaker vector is nonzero at the identity by the Casselman- 
Shalika formula. 

Suppose that tt = (8)t,7r„ is an automorphic self-dual cuspidal represen- 
tation of GL„(A) where the restricted tensor product is taken with respect 
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to a choice of unramified vectors almost everywhere. We choose invariant 
positive definite Hermitian forms (•, Ott^ '^v for all v so that (e„,e^),r„ = 1 
almost everywhere. This gives rise to sesqui-linear forms (•, ■)y^s '■ ^{t^v^ ~s) x 
/(vTtjjS) ^ C as above. We have (•, •)7r = c (g) and {•,-)s = c(8)(-,-)^^s in 

the obvious sense, for some positive scalar c, and l-j^ — ^v^-k^' 

At this point it is useful to normalize M^i^s) by the normalization factors 



m. 



TTt,, s) = my{s) defined by Shahidi in [Sha90b]. The latter are given by 



L(2s,%v,sym^) G = S0(2n + 1) 

L{s,iXy) L{2s,-Kv,K^) G = Sd 

e{s,'Kv,il>Z^)L{s+\,'Kv) £(2s,7r„,A2,'!/'ir^)i(2s+l,7r^,A2) 
L(2s,7r„,A^) 



G = S0(2n), 



e(2s,7r„,A2,i/'t, )L(2s+l,7rt, .A^) 

where L(s,7r^), L(s, tt^,, A^), L(s, vr^, sym^) are the local L-functions pertain- 
ing to the standard, symmetric square and exterior square representations of 

GL„(C) respectively, and similarly for the epsilon factors. We write M.y(-7r.y, s) = 



'4>v, 



TT-u, s)Rf^ {tTv, s) where i?u(s) = i?^"" (vr^,, s) are the normalized intertwin- 
ing operators. Note that by changing tp^ to tpvid') the scalar my{s) is multi- 
plied by (u;7r„(a) |a|"^*~2)^'= where fc = n -|- 1, n, or ra — 1 according to whether 
G = S0(2n + 1), Sp„ or S0(2n). 

The following lemma will be proved in the next section, together with the 
other lemmas below. 

Lemma 1. For all v, i?„(s), My{s), L^(2s, vr^, sym^), Lt,(2s, 7r„, A^), 
Ly{s,Trv) and m„(s) are holomorphic and nonzero for Re(s) > ^. 

In fact, the holomorphy and nonvanishing of Rv{s) for Re(s) > 5 is proved 
more generally in a recent paper of Kim ( [Kim02] ) . 

Let m{s) = m{-K, s) = m^"(7r^, s) and R{s) = ®yRy{s) so that M(s) = 
m{s)R{s). If G = S0(2n + 1) then 

L(2s, vr, sym^) L(l — 2s, vr, sym^) 

TTliS] = = — ^ 

^' e(2s,7r,sym2)L(2s-hl,7r,sym2) L(l -h 2s, tt, sym2) ' 
If G = Sp„ then 

L(s,7r) L(2s,7r,A2) 



m{s) 

If G = S0(2n), 

m{s) 



e(s, 7r)L(s + 1, vr) e(2s, vr, a2)L(2s + 1, tt, A^) 

L(l-s,7r)L(l-2s,7r,A2) 

L(l + s,7r) L(l + 2s,7r,A2)' 

L(2s,7r,A2) L(l - 2s,7r, A^) 



e(2s, TT, a2)L(2s + 1, TT, a2) L(1 + 2s, tt, A2) ■ 
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In particular, the residue m_i at s 



reSs=iL(3,7r,sym^) 
£(l,7r,sym^)L(2,7r,sym^) 



e(i,7r)L(|,7r)£(l,7r,A^)L(2,7r,A^) 

reSs=iI/(.s,7r,A-^) 
e(l,7r,A^)L(2,7r,A^) 



is equal to 2 times 

G = S0(2n + l) 
G = Sp„ 
G = S0(2n). 



By Lemma 1, tt is of G-type if and only if m{s) has a pole (necessarily simple) 
at s = ^. Thus, TT is of Sp^ type if and only if tt is symplectic and L(i, tt) 7^ 0; 
TT is of S0(2n + 1) type if and only if tt is orthogonal; tt is of S0(2n) type if and 
only if TT is symplectic. Suppose that tt is of G-type. Let 5S(s) = ^{tt,s) be 
the operator l{ — s) o R{s) : /(vr, s) I{tt, —s) for s € M and let J(7r, s) be the 
form on /(vr, s) defined by (*8(s)(/9, Since = m_i • *B (^^j it follows 

from (1) that 3(tt, ^) is semi-definite with the same sign as m_i. We will show 
that 



(4) 

and thus 



3{tt, -) is positive semi-definite 



(5) 



m_i > 0. 



2.1. Proof of Theorem 1. We will use (5) for the groups Sp„ and S0(2n). 

Together, this implies that if tt is symplectic and L{^,tt) 7^ then ^^i^^^^^a ^-^ 

> 0. By the functional equation and the fact that L{^,tt) / we must have 
e{^,TT) = 1. On the other hand L{s, tt) is a convergent Euler product for s > 1, 
all factors of which are real and positive. Indeed, L{s, tt^) = L(s, TTy) since tt^ is 
equivalent to its Hermitian dual. In the nonarchimedean case, L(s, tt^) ^ 1 as 
s — > -I-CX3 (s real). In the archimedean case L(s, TTy) = n?=i ^M.is — Si) for some 
SieC where r^{s) = 7r-*/2r(s/2). We have Elm^i = 

since TTy = TTy. It is 

easily deduced from Stirling's formula that L{s,tTv) — -|-oo as s — >■ -|-co. In 

both cases L{s,TTy) is holomorphic and nonzero for s > ^. The claim follows. 
Hence L(|,-7r) > 0, and therefore, L{^,tt) > 0. 

It remains to prove (4). The operator ^{tt, s) and the form 3(tt, s) admit 
a local analogue and we have Q5(7r, s) = (8)^05^" (vr^, s) and 3{tt,s) = c 0y 

'3^'-{TTy,s). 

Wc will prove the following purely local Lemmas. Recall the assumption 

that 6 = 1. 

Lemma 2. Let TTy he a generic irreducible unitary self -dual representation 
o/GL n over a local field of characteristic 0. Then !B^"(7r^,, s) is Hermitian for 
s G M and holomorphic near s = 0. Moreover., 'B'^" (vr^, 0) is an involution with 
a nontrivial +l-eigenspace. 
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Lemma 3. Under the same assumptions, suppose further that ^^''(vr^, ^) 
is sem,i- definite. Then Df^''(7r^,0) is definite with the same sign as ^). 
Hence, by Lemma 2, QS'^''(7r„, 0) = 1 and J'^''(7r„, ^) is positive semi-definite. 

These two lemmas, together with the fact that is semi-definite, 

imply (4), even locally. 

We remark that in the case where G is an orthogonal group then up to a 
positive scalar ^B'^" (Tr^,, s) is independent of ij^y. This is no longer true in the 
Spn case if the central character of 7r„ is nontrivial. In that case. Lemma 2 
actually implies the well-known fact that /(7r„,0) is reducible. 

Note also that the very last (and most important) conclusion of Lemma 3 
is trivial in the unramified case. Finally, let us mention that a property related 
(and ultimately, equivalent) to the conclusion of Lemma 3 for the local com- 
ponents of a symplectic cuspidal representation was proved by Jiang-Soudry 
using the descent construction ([JS]). We will not use their result. 

2.2. Proof of Theorem 3. We first observe that L(s,7r,sym^) and 
L(s, 7r,A^) are holomorphic and nonzero for Re(s) > 1. Indeed, the partial 
L-functions L^{s, vr, sym^), L'^(s, vr, A^) are holomorphic for Re(s) > 1 ([JS90a], 
[BG92]) and their product is {s,7r^7r), which is nonzero for Re(s) > 1, since 
the Euler product converges absolutely ([JS81]). The statement now follows 
from Lemma 1. 

Suppose that vr is orthogonal. Applying (5) to the group S0(2n + 1) 
we obtain T^ess=iL{s,n,syra ) ^ Since L(s, vr, sym^) is real and nonzero 

for s > 1 we obtain '''^'^ j^(2^^!,ym^)™ > 0. Hence e(l,7r, sym^) > 0. Since 
e(s,7r, sym^) is nonzero and real for s € M we get e(^,7r,sym^) > 0. On 
the other hand, e(i,7r, sym^) = ±1 by the functional equation and hence, 
£:(|,7r,sym^) = 1. Similarly, if tt is symplectic then using the group G = 
S0(2n) and the same argument we obtain e{^,Tr,A^) = 1. Since any self- 
dual cuspidal representation vr is either symplectic or orthogonal, the above 
argument shows that either £(i,7r, A^) = 1 or £(|,7r,sym^) = 1. On the other 
hand for any tt (self-dual or not) 

(6) e{s, vr (8) tt) = e{s, tt, A'^)e{s, vr, sym^). 

Indeed, this follows from the corresponding equality of L-functions, which is 
in fact true locally. In the archimedean case this follows from the compati- 
bility of L-factors with Langlands classification ([Sha90b]). For p-adic fields 
this is Corollary 8.2 of [Sha92] in the square- integrable case and follows from 
multiplicativity ([ShaQOa]) in the general case. Note that on the left-hand side 
we may take the cpsilon factor as defined by Jacquet, Piatetski-Shapiro and 
Shalika ([JP-SS83], [JS90b]); it coincides with the one defined by Shahidi; see 
[Sha84] . To finish the proof of Theorem 3 it remains to note that £(^ , 7r(8>7r) = 1 
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for any cuspidal representation vr of GL„(A). This follows at once from the 
next lemma which, at least in the nonarchimedean case, was proved (even 
without the genericity assumption) by Bushnell and Henniart ([BH99]). 

Lemma 4. For any generic representation tt^ of GL„ over a local field of 
characteristic 0, 

(7) e Q,7r„ (g)7f;,V'^^ = a;^„(-l)"~\ 

where u-,^^ is the central character of tt^ . 



3. Local analysis 

In this section we prove Lemmas 1-4 which were left out in the discussion 
of the previous section. 

For the rest of the paper let F be a local field of characteristic 0. We 
will suppress the subscript v from all notation and fix a nontrivial character 
ip oi F throughout. As before, the F-points of an algebraic group X over 
F will often be denoted by X. We denote by v the absolute value of the 
determinant, viewed as a character on any one of the groups GL„(F). If tt is 
a representation of GL„ and s G C we let ttu^ be the representation obtained 
by twisting vr by the character u^. Let Irr„ be the set of equivalence classes 
of irreducible (admissible) representations of GL„. Given representations tt^, 
i = 1,. . . ,k of GL„. we denote by vri x . . . x vr^ the representation on GL„ 
with n = ui + . . . + Uk induced from the representation tti (8) • • • <8) tt^ on the 
parabolic subgroup of GL„ of type (ni, . . . ,nfc). 

3.1. Proof of Lemma 4. For completeness we include a proof which was 

communicated to us by Hcrvc Jacquet. We arc very grateful to him. 

By the functional equation the left-hand side of (7) is ±1. We prove the 
lemma by induction on n. If tt is not essentially square-integrable then we can 
write TT = TTi X 7r2 where tt^ G Irr„. are generic. We have 

e Q,7ri ^7ri,il?je Q,7r2 (8) ttI, V' 

= £ Q , TTl ^, £ Q , 7r2 ® ^, V^) < (- IX (-1) 

= <^(-iXH-i) 

by the functional equation ([JP-SS83, p. 396]) and the dependence of epsilon 
on ijj. By "multiphcativity" of epsilon factors (loc. cit., p. 452) we get 

£Q,7r(g)7f,V') =£Q,7ri®^,^^ £Q,7r2®^,V') w;?,^(-l)a;;ji(-l) 
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and we may use the induction hypothesis. Thus, it remains to consider the 
case where tt is essentially square-integrable, which immediately reduces to the 
case where tt is square-integrable. In this case the zeta integral 

^{s,W,W',^)= f W{g)W'{g)mO,...,0,l)g)\detg\' dg 

converges for Re(s) > (loc. cit., (8.3)). Here W, W are elements in the 
Whittaker spaces of vr and vf respectively, and $ is a Schwartz function on F". 
In particular, L{s, vr (g) tt ) has no pole (or zero) for Re(s) > and by the local 
functional equation (loc. cit., p. 391) we get 

(8) ^Q,i^,W,l>) =£Q,7r®7f,v)a;,(-l)"-'^'Q,W^,W,$) 

for any W and $. Choose W ^ and let g be such that W{g) ^ 0. We may 
choose ^> > such that 'I>((0, • • • , 0, l)*;) 7^ and $ > 0. For example, we may 
take $ of the form where $1 > 0. Then clearly, both zeta integrals 

in (8) are nonnegative and the one on the right-hand side is nonzero. Hence 
£(5, TT (8) TT, V') has the same sign as a;7r(— 1)""^ and consequently, it is equal to 
it. This finishes the proof of Lemma 4 

If TT G Irr„ we denote by e(7r) the (central) exponent of tt. It is the 
unique real number so that iri^^'''^'^^ has a unitary central character. If vri, 7r2 
are generic and irreducible we let 7V4(7ri,7r2) be the normalized intertwining 
operator tti x 7r2 — > 7r2 x tti (depending on ip) as defined by Shahidi ([Sha90b]) 
provided that it is holomorphic there. 

We recall that if tt and vr' are essentially square-integrable and |e(7r) — e(7r')| 
< 1 then TT X tt' is irreducible and vr x vr' ~ tt' x vr. 

Recall the classification of the irreducible generic unitarizable representa- 
tions of GL„. (This is a very special case of [Tad86] in the p-adic case and 
[Vog86] in the archimcdean case; cf. [JS81] for the unramified case.) These are 
the representations of the form 

(9) (Ti X . . . X (Ts X Tiv'^^ X Til'~'^^ X ... X Ttv'^* X Tfl''"^* 

where the crj's and the Tj's are square integrable (unitary), the crj's are mutu- 
ally inequivalent and < 7^ < ^. Moreover, the data {o'i)i=i, {Tj,^jYj^i are 
uniquely determined up to permutation. Clearly, vr is self-dual if and only if 
{ai, TjV^^} = {(Tj, fji''^^} as multi-sets. Let H^-'i-"- be the set of self-dual generic 
irreducible unitarizable representations of GL„. 

Let S = {Sn}n>o be any one of the families B = S0(2n + 1), C = Sp„ or 
V = S0(2n) (with So = 1). The family will be fixed throughout. In each case, 
except for SO (2), the group G = Sn is semisimple of rank n and we enumerate 
its simple roots {ai, . . . , a„} in the standard way. Recall the automorphisms 9 
and S of G defined in the previous section. If vr is a representation of G we let 
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6(7r) be the representation obtained by twisting by 6. Similarly for T,{n). We 
let Irr(S'„) be the set of equivalence classes of irreducible representations of Sn- 
Let TTi, i = 1, . . . ,k, be representations of GL„- and a a representation of Sm- 
Let n = ni + . . . + rifc + m and Q be the parabolic subgroup of Sn obtained 
by "deleting" the simple roots a„j , ani+n2 > • • • > «ni+...+nfc , as well as a„ in the 
case where S = V and m = 1. The Levi subgroup L of Q is isomorphic to 
GL„^ X ... X GLnk ^ ^m- As in [Tad98] we denote by tti x . . . x TTfeXcr the 
representation of Sn induced from the representation tti ® . . . (^i) ® a oi Q. 
We have, vr x rxcr = vrxi (rxcr). In the case S = V we have S(7rx(7) = TrxT,(a) 
for TT G Irr„ and a G Irr(5m) with m> 1. 

Let L be a Levi subgroup of G and let wq (resp. 'u;q ) be the longest 
element in the Weyl group of G (resp. L). We denote by wl the Weyl group 
element wqWq. In particular wm is defined, where we recall that M ~ GL„ is 
the Siegel Levi. 

Suppose that TTj G Irr„. are essentially square integrable with e(7ri) > 
0(712) > . . . > c(7rfc) > and that a G IxY^Sm) is square integrable. Let Q and 
L be as before. Then 

L TTi X . . . X TTfcXcr admits a unique irreducible quotient. 

2. The multiplicity of this quotient in the semi-simplification of tti x . . . x 
TTfcXcr is one. 

3. The quotient is isomorphic to the image of the (unnormalized) intertwin- 
ing operator 

: TTi X . . . X vTfcxa S"i+-+"'=(7ri^ x . . . x TTfe^xcr) 
with respect to Q and w where w = wl- 

4. My, is given by a convergent integral. 

This is the Langlands quotient in this setup. For all this see [BWOO]. Let Q' be 
the parabolic subgroup with Levi subgroup L' isomorphic to GL„^+. x Sm 
and let TT = TTi X . . . X TTfe. The operator My, is obtained as the composition of 
the intertwining operator 

(10) TTXCT ^ S"l + -+"'=(7r^X(T) 

with respect to Q' and Wl', and an intertwining operator M2 "inside" 
Ghni+...+nk- Under the weaker hypothesis that c(-7ri) > ... > ^[71^) > 
the statements 1-3 will continue to hold provided that M2 is normalized. This 
is because the i?-groups for general linear groups are trivial. In particular, if 
TT is irreducible then the Langlands quotient is isomorphic to the image of the 
intertwining operator (10). 
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If TT is a representation of GL„ we let /(tt, s) = /'^(vr, s) be the in- 
duced representation ttv^xI. Similar notation will be used for induction from 
the parabolic subgroup 0{P)- We denote by M(7r, s) = M{s) : I{n,s) 
/(vr",— s) = 9{I{tt^ , —s)) the unnormalized intertwining operator with respect 
to P and wm- If ti" is generic we denote by R{Tr,s) = R{s) the normalized 
intertwining operator (with respect to P and wm)- In the case G = SO (2) we 
set M{s) = R{s) = 1. 

We will often use the following fact. Suppose that vr = vri x . . . x vr^ is a 
generic representation of GL„ with VTj G Irr„.. We may identify /(tt, s) with 
IndQ(7rii/*' (g) . . . (g) TTfcZ^'^) where tti (g . . . (8) vr^ is viewed as a representation of the 
parabolic subgroup Q of G whose Levi subgroup is the Levi subgroup of GL„ 
of type (rii, . . . , n^). We may also identify tt^ with TTf} x . . . x tti^ and I{it^,s) 
with Indg^TTfeV'' (g) . . . (g) TTi^;/*. Under these identifications R{s) becomes the 
normalized intertwining operator 

Ind§(7rizy* g) . . . TTki^') ^(Indg, (TTk^u'' g) . . . (g vr/i/"^)) 

with respect to Q and wm- This is merely a reformulation of the multiplicativ- 
ity of L and £-factors ([ShaQOa]). As a result, we may decompose the operator 
i?(7r, s) as a product of "basic" intertwining operators according to the reduced 
decomposition of wm- Each basic intertwining operator is obtained by inducing 
an operator of the form R{'Ki, s) or M{'Ki,Trj) or M{iri,Trj^) with i > j. 

3.2. Proof of Lemma 1. Let vr G n^-*^'"' and Re(s) > |. We may write vrz^* 
as TTi X . . . x TTfe with vTj essentially square-integrable and e(7ri) > . . . > e(7rfc) > 
0. Hence /(tt, s) admits a Langlands quotient, which by the discussion above, 
is given by the image of M(7r, s). By multiplicativity, the statements about the 

L-functions follow from the holomorphy of L{s,TTi), VTj, sym^), L(s,7rj, A^) 
and L(,s, TTj (g TTj) at s = 0, which in turn follows from [Sha90b, Prop. 7.2]. The 
statements about the normalizing factors and R{tt, s) follow immediately. □ 

3.3. Proof of Lemma 2. Recall the definition of the operators 5S'^(7r,s). 
(We assume that 6 = 1 and that vr G Irr„ is self-dual.) We first note that 
ijr is Hermitian since, being an intertwining operator of order two, it must 
preserve the inner product. We conclude that i* ^ = where * denotes the 
Hermitian dual. Also, a direct calculation shows the relation 

M(7r«,s)6^8_, = 6^_,M(7r,s). 

Moreover, by (2) the Hermitian dual of M{it,s) is given by ujT^{—l)^M{7r'^,s) 
where A; = n if G is symplectic and /c = n + 1 if G is orthogonal. On the other 
hand, by the dependence of root numbers on the additive character it is easily 
deduced that 

m^(7r,s) = u;^(-l)*^m'^(7r, s). 
The Hermitian property of ^'^{tt, s) for s real follows. 
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To prove the second part we use the argument of [KS88, Prop. 6.3]. 
Let Wf{-,s) be the Whittaker functional on /(tt, s) and let W^j|(-,s) be the 
Whittaker functional on /(tt^js) obtained through l.^. They are holomorphic, 
nonzero ([ShaSl]), and satisfy the functional equation 

(11) W,^(^,s) = c(7r,s,V')WjJ(M(7r,s)^,-s) 

where c'^(7r,s) is the "local coefficient" which was studied by Shahidi. By 
[Sha91, Th. 3.5] it is given by 

L(^2s,7r,r) ^> - 15, U 

e{s,n,'^~^)L(l—s,Tr) £(2s,7r,r,i/)~^)L(l— 2s,7r,r) o 

L(i^ ■ L{2s,TT,r) O—U 

where r = sym^ for 5 = ;S and r = for <S = C, V. (Here we use that tt is 
self-dual.) By the identification : tt'^ ^ tt, (11) becomes 

Wtiv, s) = c'^(7r, s)m^(7r, s)Wf{^'^{TT, s)ip, -s). 

The term c'l' {tt , s)m'^ {tt , s) is either L(l - 2s,7r,r)/L(l + 2s,7r,r) ii S = B,V 
or L(l - 2s, TT, r)/L(l + 2s, TT, r) ■ L(l - s, 7r)/L(l + s, tt) if 5 = C. It follows 
that 

Q5'^(7r,-s)*B'^(7r,s) = 1. 

We infer that Q3^(7r, s) is unitary, and in particular, holomorphic at s = 0. 
Moreover, *B'^(7r,0) fixes the i/^-generic irreducible constituent of /(7r,0), since 
L{s, tt) and L(s, tt, r) are holomorphic at s = 1 by Lemma 1. 

The rest of the paper is devoted to the proof of Lemma 3. Since the lemma 
is evidently independent of the choice of the character ^p, we will suppress it 
from the notation. 

3.4. Representations of G-type. Let cr be a self-dual square-integrable 
representation of GL„ and suppose that 9 = 1. By the theory of i?-groups 
(e.g. [Gol94]) the following conditions are equivalent. 

1. /(cr, 0) is irreducible. 

2. 03 (cr, 0) is a scalar. 

3. The Plancherel measure //(a, s) is zero at s = 0. 

Definition 1 . An essentially square-integrable representation cr of GL„ will 
be called of G-type (or of S-type if we do not want to specify n) if it is self-dual 
(in particular, e(cr) = 0), = 1, and the conditions above are satisfied. 

Proposition 1. Let a he a square-integrable representation of GL„. 
Then I^{a, s) is irreducible for < s < 1 except possibly for s = ^. Moreover, 
if I {a, ^) is reducible then a is of G-type. 



(12) 



'(7r,s) 
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Proof. By the results of Muic ([MuiOl]) we may use Proposition 5.3 of 
[CS98]. Thus the reducibihty points of I{a,s) for s > are the poles of 
L(l - 2s, a, r) {if S = B or V) or L(l - s, a)L{l - 2s, a, r) (if S = C). These 
L-functions are computed in [Sha92, Prop. 8.1]. In particular, L{s,a,r) is 
holomorphic for s > — 1 except possibly for s = and L{s,a) is holomorphic 
for s > 0. Therefore I{a,s) is irreducible for < s < 1 except possibly for 
s = 5 and moreover, if I{a, ^) is reducible then L{s,a,r) has a pole at s = 0. 
In the latter case 9 = 1, a is self-dual and the local coefficient vanishes at 
(loc. cit.). By [ShaOOb, (1.4)] the same will be true for the Plancherel measure. 

□ 

Remark 1. Shahidi also proved the following in ([Sha92]). Suppose that a 
is a self-dual square-integrable representation of GL„ which is not the trivial 
character of GLi. Then the following are equivalent: 

1. a is of Sp„ type. 

2. a is not of S0(2n -|- 1) type. 

3. a is of S0(2n) type. 

In particular, in this case n must be even. We will not use this fact. 

For convenience, we consider the set n*^ *^' of all representations of the form 

(13) ai X . . . X as X Ti X Ti" X . . . X Tt X 

where the (jj's are square-integrable, self-dual and (as we may assume) mutually 
inequivalent, and the Tj's are essentially square-integrable with < c^tj) < ^. 
Any element of n** *^' is irreducible, generic and self-dual. Clearly, n'^ '^' D 
jjs.d.u. rpj^g condition on tt G n^ d- iq belong to n^ *i "- (i.e. to be unitarizable) 
is that each tj which is not square-integrable appears in (13) the same number 
of times as tj'^i/'^^^'^^'K If tt G Jis.d. |^]^gj^ |-,y ^]^g discussion of subsection 3.2, 
/(tt, ^) admits a Langlands quotient, which will be denoted by £Q(7r). It is 
obtained as the image of -R(7r, |) (or M(7r, |)). 

Also, if X is an essentially square-integrable representation of GL„ we de- 
note by SP{x) the unique irreducible quotient of xi/2 x X'^'^- It is isomorphic 
to the image of the intertwining operator Ai{xi^^ ,X^~^)- 

Lemma 5. Let x an essentially square-integrable representation of 
GL„ with < e(x) < i. Assume that x is not of S-type. Then CQ(x x x'') — 
E-(5P(x)xl). 

Proof. The Langlands quotient is obtained as the image of the longest 
intertwining operator, which is the composition of the following intertwining 
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operators: 

( X X X , 2 ) ' Xi^2 xiS" (^XJ^ 2 xlj ~ E" [xvi x 2 xlj 

>Tr{x^ 2 X xz^2 xl 1 ~ x^^ 2xEMx^2x'l) 

lxE"(il(x,|)) _i b -i 1 

> X^ ^ X 

where i?i = A4(x'^^5X^^~^)- By Proposition 1 the only map which is not an 
isomorphism is E"(/(i?i, 0)), whose image is E"(5P(x)xil) as required. □ 

Any TT G n^-"^- can be written uniquely as j^^^^^S-^yv^ x t^^'^^S-p^^"^^ x 

^pur^5-type ^j^jj 

« ^purc-5-typc q£ form o"! x . . . x (T5 where the aj's are square-integrable, 
self-dual and of 5-type; 

• 7]-"°'i-'5-typc of the form pi x . . . x where the pi's are square-integrable, 
mutually inequivalent, self-dual and not of 5-type; 

^ ^non-5-pairs q£ form Ti X Ti'' x . . . x X Tf' whcrc thc Tj's are essentially 
square-integrable, not of 5-type (self-dual or not), and < e(rj) < ^. 

Note that n'^'^'^-^-^yp^ and 7rP"''^'5-*yP'^ are tempered. 

Definition 2. We say that tt G H^'^- is of G-type if tt'^^'^-^-^^p^ = 0. 

The definition is suggested by the local Langlands reciprocity. Note that 
if TT is of S0(2n) type then n is even. 

The crucial property of representations of G-type is the following. 

Lemma 6. Ifir^ jjs.d. of G-type then ©(0) is a nonzero scalar. 



Proof. Wc use induction on n, the case n = being trivial. For the 
induction step, we can assume that tt = vr' x u; where vr' G Y\^ <^- ig of (S-type 
and u G Irr; is either square-integrable and of <S-type or of the form txt^ where 
r G Irr^ is essentially square-integrable. Note that I is even if 5 = X*. The 
operator i?(0) can be written as the composition of the following intertwining 
operators: 

(14) /(tt' X a;, 0) : /(vr' x a;^ 0) : 

I{J X 7r',0) '^"^^^''"^ > liJ X Tr'\0) ~ /((tt' x u;)\o) 

where Ri = M{Tr' ,uj^). The last identification is induced by the isomorphism 
uj'" X tt''' ~ (tt' X a;)^ By the induction hypothesis the third map is a nonzero 
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scalar multiple of lxit(7r', 0)-^ Also, by uniqueness, : u)' y. it''" tt is a 
scalar multiple of (1 x Lij)R^^{l x t^). All in all, the map ©(vr, 0) is a scalar 
multiple of 

/((I X t^)R{^{l X i;),0) o lxi(7r',0)-^ o/(i?i,0) o lxi?(u;,0) = lx«B(a;,0). 

It remains to show that *8(u;,0) is a scalar in the two cases above. In the 
first case, this follows from the definition of 5-type. In the second case, we 
decompose R{u!, 0) as before as 

I{r X r^0) ^^^1^^ T.-{I{r x r,0)) ^1:^^ 

E-(/(r X r, 0)) /(r x r^ 0) ^ /((r x , 0). 

Note that the map R2 = M{t, t) is a scalar, and similarly for the map t,^ : t x 
~ (r X T^f ^txtK Thus, *B(a;, 0) is a scalar multiple of lxI]"*(i?(T, 0)) o 
R{t^,0) which is 1 by the properties of the normalized intertwining operator. 

□ 

Remark 2. The converse to Lemma 6 is also true. 
3.5. Langlands quotient. 

We extract a few results from [MW89] (cf. 1.6.3 for the p-adic case and 1.7 
for the archimedean case). 

Lemma 7. Let n and ir' be irreducible representations of GL„ and GL„/ 
respectively. 

1. //tt X tt' is irreducible then tt x tt' ~ tt' x tt. 

2. Let TT and tt' be essentially square-integrable. Suppose that |e(7r) — e(7r')| 
< 1. Then tt x tt', ttu^ x SV{Tr') and SV{'it) x SV{'it') are irreducible. 

3. Suppose that tt and n' are inequivalent square-integrable representations. 
Then ttv'^ x 7rV~2 irreducible for — 1 < 7 < 1. 



We will also need the following lemma which is based on [Jan96] . 

Lemma 8. Let iTi G Irr„. for i = 1, . . . ,k and a G Iii{Sm)- Suppose that 
TTi X TTj, TTj X tt/ are irreducible for all i ^ j and iViXia is irreducible for all i. 
Then 

(15) TTi X . . . X TTfeXa ~ x . . . x TTk^Xa). 

Suppose in addition that the i^iS are essentially square-integrable with ^{'Ki) > 
and o is square-integrable. Then tti x . . . x WkXa is irreducible. 
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Proof. In the case where A; = 1 we note that if tt G Irr„ and a € Irr(5m) 
then TT XI cr = i;"'(7r'' XI cr) in the Grothendieck group since tt cr and S"(7r'' (g) a) 
are associate. The case k > 1 and the last statement are proved in ([Jan96]) for 
the cases S = B,C. The proof carries over almost literally (except for putting 
in some S's) to the case S = T> (cf. Proposition 2 below). □ 

Let vr € W""^'. Recall that /(vr, ^) admits a Langlands quotient, denoted 
by CQ{k), which is isomorphic to the image under i?(7r, ^). 

Proposition 2. Zei tt = T^^'^^s-tyv^ x ;r"°"-'^-P^'''^ x TrPu^e-^-type ^ jj^.d. 
as above. Then 

(16) /:Q(^) ~ S^(5P(ri) X . . . x5P(rt) x^'^°'^-'5-*yPV5 x£Q(7rP"'''^-'5-tyP^)) 
for £ either or 1 {depending only on T^^°^-^-P'^^^>^"j _ Hence, 

Proof. Clearly, the second statement follows from the first. Let A be the 
right-hand side of (16). Following the argument of [Jan96, Th. 3.3] we will 
argue that 

(17) A is a quotient of /(tt, ^) for e either or 1. 

(18) A is irreducible. 

The first statement is proved by induction on n, as in subsection 3.4. Since the 
case where -j^^o^S-pairs = q is immediate, we may assume for the induction step 
that TT = tt' X T X where vr' G W '^-, r G Irr^ is essentially square-integrable, 
< e(r) < ^ and r is not of 5-type. 

It follows from Lemma 5 that up to S, /(vr, ^) has a quotient isomorphic 
to /(ttVs X SV{t),0). It follows from part 2 of Lemma 7 that tt'u^ x SV{t) 
is irreducible, and hence, that Tr'u2 x SV{t) 2± SV{t) x TrVa. We deduce that 
up to S, /(tt, \) admits I{SV{t) x 7rV2,0), and thus also (S7^(r)x>CQ(7r'), as 
a quotient. This implies (17) by the induction hypothesis. 

To prove (18), it suffices to show that ^(A) ~ A. (This condition does 

not depend on e.) Indeed, wc have 9[CQ{tt)) ~ CQ{tt) (cf. [Jan96]) since both 
sides are the unique irreducible subrepresentation of /(tt, — |) by (3). We would 
conclude that CQ{tt) is both a quotient and a subrepresentation of A. However, 
CQ{tt) is the unique irreducible quotient of A, and it has multiplicity-one in 
the semi-simplification of A. Thus, A ~ CQ{tt). 

We shall write tt^ for 7rP"'"'^"'5-type ^ 'p^ show that 0(A) ~ A we note once 

more that CQ{TT-i) = CQ{TT-i). By Lemmas 7 and 8 it suffices to show that 
both xiCQ[TTz) and (SP(T)x£Q(7r3) are irreducible where p G Irr; is square- 
integrable self-dual not of <S-type and r is as before. 
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To prove this, consider the representation vr' = r x x tts. The Langlands 
quotient of /(tt', ^) is the image of the operator Myj^ which is the composition 
of the intertwining operators 

/ (tU^ X t'u^ X TTsU^ , 

— y I {tu^ X tt^u^ xr^z/5,0^ 

> T,'^ (^I (tV^ X TV'^ X TTsV^ ,0^^ 

— > E"" (/ (rz/"5 X Ti^^ X 7r3''z/-^,0)) 
— y E"'(/(rz/-5 x7r3^z/-5 XTi/io)) 
— y I (^Tu'^ X 7r^^u~^ X T^v~^ ,0^ 

y I (^TV~^ X T^U^^ X TT^'^U^^ ,0^ 

> I (t^U~^ X TU~^ X '!rs'u~^,0^ . 

Again by Lemma 7 and Proposition 1, all arrows except the fourth one are 
isomorphisms. Thus, the Langlands quotient is isomorphic to the image of the 
fourth map, which is S™'(5'P(r) x/IQ(7r3)). Hence, the latter is irreducible. 
Similarly, if vr' = p x 7r3 then CQ{i:') is the image of the composition of the 
intertwining operators 

/ X 7r3i^2 ,0^ 
— y I {j)u^ XTT^v^^i^Q^ 
— y I {tt^v'^ X pv^jO^ 

(/ {tts^u-^ X p^z/-i,o)) . 

Again, all maps except the first are isomorphisms. Thus, as before, pu's x£Q(7r3) 
is irreducible. □ 

For future reference, let us reformulate the conclusion of Proposition 2. 
Using a decomposition of wq we may decompose R{tt, 5) as 

(19) 

^ (/ (X^'^i (r^i/^ X TjZ/-^) X 7r°°°-'5-*yPV^ x TrP^^'^-^-^^PV^, o)) 
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X TT 



.non-5-type'' -i 



1/ 2 X tt' 



.pure-<S-type 



z/-2,0 



I) 



where R^i are normalized intertwining operators. We observe that the image 
of (of the whole induced space) is isomorphic to the right-hand side of (16), 
and hence it is the Langlands quotient. By irreducibility and multiplicity-one of 
Langlands quotient im(i2„,2 o Rwi) = i™(-fi^2) ker^R^s ° Rw2) = ker(i2„,2). 

3.6. Reduction to the tempered cxise. Let vr e n'-'i-u-. We may write 
TT = Trtcmp ^ ^n.t. ^j^gj-g y^tcmp ^ -Qs.d.u. tempered and vr"^-*- is of the form 

Xj(a;ji^^' X Ui'v'^^) with square-integrable and < /3i < ^. Clearly, vr"'*' 
appears as a factor of Tr^o^-'^-pairs ^-^^ deform the nontempered parameters 
of TT. For < t < 1 let 



Then ttj is a "deformation" in n^ d u- from tt ~ tti to the tempered representa- 



tion TTo. Clearly t,-^'"'-^-^^ = ^^non-S-type foj. all t and Trf""^^^"*^" = TrP^'^^'S-tyP^ 



for t ^ although not necessarily for t = 0. The form J(7r, s) depends on 
the unitary structure on vr, or what amounts to the same, on a GL„-invariant 
positive-definite Hermitian form on vr. We identify the ambient vector spaces of 
TTt with that of tt in the usual way. The X-action does not depend on t, where 
K denotes the standard maximal compact. We may choose a family of GL„- 
invariant positive-definite Hermitian forms on nt which depends continuously 
on t (using intertwining operators for example). 

The following lemma will reduce Lemma 3 to the tempered case. 

Lemma 9. 1. The definiteness o/2J(7rt,0) does not depend on t. 

2. If3{'!r, ^) is semi-definite then J(7ro, ^) is semi-definite with the same 



We will use the following elementary lemma. 

Lemma 10. Let {lp}a<f3<b be a continuous family of Hermitian forms 
on C™". Suppose that rank(/^) is constant for a < (3 <h and that If, is positive 
semi- definite. Then la is positive semi- definite. 

Indeed, both parameters of the signature (s+(/3), s_ (/?)) of Ip are lower 
semi-continuous functions. By the conditions of the lemma, s+(/3) -|- s_(/3) is 
constant on (a, b], and hence the same is true for s±{f3). 

Proof of Lemma 9. Since R{iTt, 0) is invertible, 3i(7rt, 0) is a nondegenerate 
Hermitian form on /(7rt,0) for any t. Thus, the first statement follows from 
Lemma 10, after passing to any if-type. 




sign. 
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To prove the second part, we will apply the discussion following Proposi- 
tion 2 to the representations ttj. We may identify all the induced spaces in (19) 
with the ones for t = in the usual manner. The JC-action will be independent 
of t. Wc obtain a decomposition of the operator i{iTt, —^)R{'iTt, ^) defining the 
form 3(TTt, ^) as Ct o B o At such that for i 7^ we have im(i? o At) = im(i?) 
and ker(Ct o B) = ker(S). The crucial point is that the operator B (denoted 
by in (19) does not depend on t. Thus on each K-type of /(ttj, ^) the rank 
of 3(vrt, ^) is equal to the rank of B, as long as t 7^ 0. Thus, we may apply 
Lemma 10 to conclude the second statement of the lemma. □ 

3.7. The tempered Case. We continue the proof of Lemma 3. By virtue 
of the last section, we may assume that vr is tempered. In this case, the repre- 
sentations /(tt, s) are irreducible for < s < ^ by Lemma 8 and Proposition 
1. Thus 3{Tr,s) is nondegenerate for < s < ^. We will show below that 
if ^(vr, i) is semi-definite then tt is of G-type. Then by Lemma 6, ^(vr, 0) is 
definite. We may use Lemma 10 on each i^'-type to conclude Lemma 3. 

It remains to show that tt is of G-type if tt G n'^-d-u. jg tempered and J(7r, ^) 
is semi-definite. To shorten notation, let tti = 7r"°'^-'5-paire x 7rP"'^e-'5-type g n^ '^ "- 
and 7r2 = 7rnon-5-typc ^^iat tt = vri x 7r2. Note that tti is of G-type and hence 
*B(7ri, 0) is a scalar by Lemma 6. Since /(tti, s) is irreducible for < s < ^ it 
follows from Lemma 10 that 

(20) ^(^^'^) is semi-definite. 

We need to show that 7r2 = 0. Consider the family 

I ^TTi (8)772, := I (ttiz^^ X 7r2^^,o) . 



Let 



OS' (7) :i"(^7ri(8)7r2, Q,7)) ^/(^7ri®7r2, 
be the operator ^(7) o 7^(7) where 

7^(7) :/(^7ri®7r2, Q,7)) ^ / (^V 7r2^ 
is the normalized intertwining operator and 



— > I (^TTi (8)7r2, y-^^-l 

As usual we identify the underlying iiT-module of each family of induced rep- 
resentations, so that it does not depend on 7. The same argument as in 
Proposition 2 with the exponent 7 > instead of \ gives: 
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Proposition 3. If j > then I (^tti ®tt2, (i^t)) admits a Langlands 
quotient which is given by the image 0/7^(7). It is isomorphic to 1:21^'^ x 

The operator TZ{'y) can be written as the composition of the intertwining 
maps 

/ ^TTl (8)772, Q,7 
1 



/ ( TTl^ (8)7r2, ( -^,7 

/ 7ri''(g)7r2% --,-7 



-4- 



2' 



where the first map is 

/(X (7r2Z/^,7ri^z/-^) ,0) o (ly^R (^1,^)) o I (7riz/5,7r2i^^) ,o) . 

As before, the intermediate map lxi?(7ri,^), which does not depend on 7, 
already gives the Langlands quotient as its image (on the full induced repre- 
sentation) for 7 > 0. Hence the rank of ©'(7) on each i^-type is independent 
of 7. Since ^'(^) = we conclude by Lemma 10 that *B'(0) is a semi- 

definite operator. 
Now, 

k(0) o lxi?(7r2, 0) = <-7rii^~2 X ^7^2 ^1 ° 1^^('^2, 0) 

= f.Tril^'"^ X*8(7r2, 0) = 1x58(712,0) O i^^l/~2 xl. 

Also, since 7r2 x 7ri^i/~2 is irreducible, 

(^i^^u~i xlj o M (7r2,7ri''z/~^^ = M (7r2, 7riz^~^^ o ^1 x l^^u'^^ 
up to a scalar. All in all, *B'(0) is equal up to a scalar to 
lxQ3(7r2,0) oi^jZy-5xlo/(A^ (7r2, 7ri^z^"^) , o) o (^Ixi? (^tti, o Mi 
= lxQ5(7r2,0) o (7r2,7riz/-5) ,0) o (^1x03 (^tti, o Mi 

where Mi = I{A4{t:iU2 , 7r2), 0). Note that by the properties of the normalized 
intertwining operators / (^M. ^7r2, 7rii^~2^ ^ 0^ is the Hermitian dual of Mi up 
to a scalar, and hence 

i(M (n2,iTiu~^) ,0) o (^1x05 (t^i^)) 
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± 



is semi-definite. On the other hand, ^(712, 0) is a Hermitian involution. Thus, 
for 05' (0) to be semi-definite it is necessary and sufficient that 

(21) Mf ^ (kei (lyiR(ni,^^^^ D ttiz/^x 

where Q,^ are the ±1 eigenspaces of Q5(7r2, 0) on I(7r2, 0). Indeed, *B'(0) is semi- 
definite, of opposite signs, on the subspaces iriU2 We will show that (21) 
is impossible if tt2 7^ 0. Let u be any irreducible constituent of /(7r2,0). The 
Langlands quotient of ttiv^-a lu is obtained as the image of the corresponding 
intertwining operator (with respect to a maximal parabolic of G) 

1 b — 1 

which is given by convergent integral. On the other hand, M2 is also the 
restriction to TTiz/axo; of the intertwining operator (with respect to a co-rank 
two parabolic subgroup of G, but the same Weyl element) 

M3 :/f7ri®7r2, f^,0) ) ^ / ( tti^ ® 7r2, ( 



2' v; V V 2 

Thus, we conclude that the image of ttiz/z xi a; under M3 is nonzero. On the 
other hand M3 is obtained as the composition of 

lUi®7r2,(^,o]] ^ ^ /r7r2®7ri,('o,^ 



lxij(7ri,i; 



/(7r2(8)7riM0,-- 



T^l (8)7r2, ( "2'^ 



1 



Thus the left-hand side of (21) does not contain ttii^z xo; for any irreducible 
constituent of I [7^2, 0). It remains to show that: 

Lemma 11. 0=^ / i/7r2 / 0. 

Proof. This follow from the theory of i?-groups (cf. [Gol94]). Indeed, let 
cr = Pi ® . ■ ■ Pr considered as a square-integrable representation of a Levi 
subgroup L of M = GLm and let Q = LV be the corresponding standard 
parabolic subgroup of Sm- Thus tt2 = Indg^pi^a. By our conditions on a, the 
i?-group of cr in Sm is isomorphic to 

W{(j) = {w e W/Wl ■■ wLur^ = L.wa ~ cr}. 

Thus any nontrivial element in W{a) gives rise to a nonscalar intertwining 
operator Ryj. Since the operator *B(7r2,0) is up to a scalar Ryj for w = wqWq 
we get the result. □ 
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Remark. Suppose that 6 = 1 and consider the fohowing conditions on a 
self-dual generic representation of GL„. 

1. TT is of G-type. 

2. *8(7r, 0) is a scalar. 

3. /(tt, ^) has a unitarizable quotient. 

4. ?8(7r, 5) is semi-definite. 

We remarked above that conditions 1 and 2 are equivalent. Similarly, con- 
ditions 3 and 4 are equivalent. In the tempered case, all the conditions are 
equivalent, although in general 3 is stronger than 1. Any local component of 
a cuspidal representation of G-type satisfies 3. It seems that 3 reflects the 
fact that TT is a functorial image of a unitarizable representation of a classical 
group. 
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